fuse: re-validate the DLM grant after waiting on the coherency gate#191
Open
hbirth wants to merge 1 commit into
Open
Conversation
Both cached IO paths request their DLM lock first and then go to sleep on things a NOTIFY invalidate can be holding: the read path blocks on the coherency gate (writer priority), the write path additionally sleeps on a contended i_rwsem. A NOTIFY invalidate running in that window revokes exactly the lock just granted (fuse_dlm_unlock_range()), so the task wakes up and populates or dirties the page cache with no DLM coverage. Close the window without ever sending a FUSE_DLM_WB_LOCK request while holding the gate (a grant that had to wait on an invalidate delivered to this same client would deadlock against our own gate hold): - Drop the lock record under the gate write side in fuse_reverse_inval_inode(), so revocation and page drop are one atomic step with respect to the gate. - After entering the gate read side, re-check the grant against the live lock tree; if it was revoked while we waited, drop the gate, re-request, re-enter and check again. With the revoke now gated, passing the check means the lock cannot go away for the whole gate hold: a revoke arriving mid-operation parks until the IO is done. - Move the write path's lock request below the inode lock, so the sleep on i_rwsem no longer sits between grant and use. This also computes the O_APPEND lock range against a stable i_size instead of a racy lockless read. fuse_get_dlm_lock() now reports whether the grant is recorded so the re-validation loop can fall through unlocked on a persistent request failure, preserving the old degraded behavior instead of spinning. Signed-off-by: Horst Birthelmer <hbirthelmer@ddn.com>
Collaborator
Author
|
@achhenderson @hazhou-ddn the key was to create a way to query whether we have a region marked as exclusive (this was not complete before since we never needed it) ... now we can operate in a loop ... if we lost the dlm lock we can reacquire it without holding the semaphore. This still has a small problem, where we probably acquire a lock twice when two writer were blocked by the notification ... but I think our code can handle that |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both cached IO paths request their DLM lock first and then go to sleep on things a NOTIFY invalidate can be holding: the read path blocks on the coherency gate (writer priority), the write path additionally sleeps on a contended i_rwsem. A NOTIFY invalidate running in that window revokes exactly the lock just granted (fuse_dlm_unlock_range()), so the task wakes up and populates or dirties the page cache with no DLM coverage.
Close the window without ever sending a FUSE_DLM_WB_LOCK request while holding the gate (a grant that had to wait on an invalidate delivered to this same client would deadlock against our own gate hold):
Drop the lock record under the gate write side in fuse_reverse_inval_inode(), so revocation and page drop are one atomic step with respect to the gate.
After entering the gate read side, re-check the grant against the live lock tree; if it was revoked while we waited, drop the gate, re-request, re-enter and check again. With the revoke now gated, passing the check means the lock cannot go away for the whole gate hold: a revoke arriving mid-operation parks until the IO is done.
Move the write path's lock request below the inode lock, so the sleep on i_rwsem no longer sits between grant and use. This also computes the O_APPEND lock range against a stable i_size instead of a racy lockless read.
fuse_get_dlm_lock() now reports whether the grant is recorded so the re-validation loop can fall through unlocked on a persistent request failure, preserving the old degraded behavior instead of spinning.